Graphical user interface and video frames for a sensor based detection system

ABSTRACT

A system may include a sensor a visual capturing device, and a controller. The sensor may be configured to measure a value associated with an input. The visual capturing device may be configured to capture visual data. The controller may be configured to cause the measured value and the captured visual data to be rendered on a display device simultaneously.

RELATED U.S. APPLICATIONS

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/281,896 entitled “SENSOR BASED DETECTIONSYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No. 13-012-00-US),filed on 20 May 2014, which is incorporated by reference herein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/281,901 entitled “SENSOR MANAGEMENT AND SENSORANALYTICS SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-013-00-US), filed on 20 May 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/315,286 entitled “METHOD AND SYSTEM FORREPRESENTING SENSOR ASSOCIATED DATA”, by Joseph L. Gallo et al.(Attorney Docket No. 13-014-00-US), filed on 25 Jun. 2014, which isincorporated by reference herein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/315,289 entitled “METHOD AND SYSTEM FOR SENSORBASED MESSAGING”, by Joseph L. Gallo et al. (Attorney Docket No.13-015-00-US), filed on 25 Jun. 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/604,472 entitled “ALERT SYSTEM FOR SENSOR BASEDDETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-015-10-US), filed on 23 Jan. 2015, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/315,317 entitled “PATH DETERMINATION OF A SENSORBASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-016-00-US), filed on 25 Jun. 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/315,320 entitled “GRAPHICAL USER INTERFACE OF ASENSOR BASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (AttorneyDocket No. 13-017-00-US), filed on 25 Jun. 2014, which is incorporatedby reference herein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/315,322 entitled “GRAPHICAL USER INTERFACE FORPATH DETERMINATION OF A SENSOR BASED DETECTION SYSTEM”, by Joseph L.Gallo et al. (Attorney Docket No. 13-018-00-US), filed on 25 Jun. 2014,which is incorporated by reference herein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/281,904 entitled “EVENT MANAGEMENT FOR A SENSORBASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-020-00-US), filed on 20 May 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/336,994 entitled “SENSOR GROUPING FOR A SENSORBASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-021-00-US), filed on 21 Jul. 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/337,012 entitled “DATA STRUCTURE FOR A SENSORBASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-022-00-US), filed on 21 Jul. 2014, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/488,229 entitled “SENSOR ASSOCIATED DATAPROCESSING CUSTOMIZATION”, by Joseph L. Gallo et al. (Attorney DocketNo. 13-023-00-US), filed on 16 Sep. 2014, which is incorporated byreference herein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/637,181 entitled “PLAYBACK DEVICE FOR A SENSORBASED DETECTION SYSTEM”, by Joseph L. Gallo et al. (Attorney Docket No.13-025-00-US), filed on 3 Mar. 2015, which is incorporated by referenceherein.

This application is a continuation in part of the U.S. patentapplication Ser. No. 14/284,009 entitled “USER QUERY AND GAUGE-READINGRELATIONSHIPS”, by Ferdinand E. K. de Antoni et al. (Attorney Docket No.13-027-00-US), filed on 21 May 2014, which is incorporated by referenceherein.

This application is related to Philippines Patent Application No.1/2013/000136, “A DOMAIN AGNOSTIC METHOD AND SYSTEM FOR THE CAPTURE,STORAGE, AND ANALYSIS OF SENSOR READINGS”, by Joseph L. Gallo et al.(Attorney Docket No. 13-027-00-PH), filed May 23, 2013, which isincorporated by reference herein.

BACKGROUND

As technology has advanced, computing technology has proliferated to anincreasing number of areas while decreasing in price. Consequently,devices such as smartphones, laptops, GPS, etc., have become prevalentin our community, thereby increasing the amount of data being gatheredin an ever increasing number of locations. Unfortunately, most of thegathered information is used for marketing and advertising to the enduser, e.g., smartphone user receives a coupon to a nearby coffee shop,etc., while the security of our community is left exposed and at a riskof terrorist attacks such as the Boston Marathon bombers. Furthermore,surveillance devices, if any, are usually a tool to bring assailants tojustice and are punitive in nature rather than being used as apreventative tool.

SUMMARY

Accordingly, a need has arisen to use a sensor based detection system todetect occurrence of a possible event, e.g., possible terroristactivity, possible radiation or bio-hazardous material, etc., andfurther to complement the sensor based detection system withsurveillance footage. There is also a need to facilitate a playbackfunctionality associated with the sensor(s) readings such that events inthe past and event in real-time can be analyzed. As a result, the sensorbased detection system equipped with a surveillance system results in apreventative system rather as well as a punitive one.

According to some embodiments, a system may include one or more sensors,and a controller. The sensor, e.g., thermal sensor, electromagneticsensor, Geiger counter, mechanical sensor, motion sensor,biological/chemical sensor, etc., may be configured to measure a value,e.g., radiation value, thermal value, electromagnetic value, etc.,associated with an input, according to some embodiments. In someembodiments, the controller may be configured to cause the measuredvalues or values derived from the measured values to be rendered on adisplay device. It is appreciated that the displayed measured values maybe a subset of all the measured values for each sensor.

According to some embodiments, the measured values may includehistorical values, real-time values, or any combination thereof. Thecontroller may be configured to display a playback control graphicaluser interface (GUI), enabling a user to scroll through time and themeasured values.

According to some embodiments, the controller is further configured todisplay information associated with the sensors, e.g., geo-locationalposition, sensor type, etc. The controller may be configured to displaya representation of the sensors and the sensor readings on a map, wherethe sensors are displayed with respect to their geo-locational positionsof one another. The controller may also be configured to display a pathtraveled by a bio-hazardous material from the measured values.

According to some embodiments, a system may include a sensor, a visualcapturing device, and a controller. The sensor, e.g., thermal sensor,electromagnetic sensor, mechanical sensor, motion sensor,biological/chemical sensor, etc., may be configured to measure a value,e.g., radiation value, thermal value, electromagnetic value, etc.,associated with an input, according to some embodiments. The visualcapturing device, e.g., video camera, digital camera, etc., may beconfigured to capture visual data, e.g., video frames, pictures, etc. Insome embodiments, the controller may be configured to cause the measuredvalue and the captured visual data to be rendered on a display devicesimultaneously.

According to some embodiments, the controller may be further configuredto cause the measured value to be time stamped and stored in a storagemedium for later retrieval. In some embodiments, the controller may befurther configured to determine whether the measured value satisfies acertain condition, e.g., whether the measured value exceeds a certainthreshold, whether the measured value falls outside of the acceptablerange, whether the measured value is within a certain range, etc.

In some embodiments, the controller in response to determining that themeasured value satisfies the certain condition may cause the capturedvisual data to be time stamped and stored in a storage medium for laterretrieval. However, it is appreciated that in some embodiments, thecaptured visual data may be time stamped automatically regardless ofwhether the measured value satisfies the certain condition.

It is appreciated that in some embodiments the sensor and the visualcapturing device are within a same physical proximity, the capturedvisual data is associated with the measured value, and the controller isconfigured to cause the measured value and the captured visual data tobe rendered on the display device simultaneously. According to someembodiments, the sensor is configured to measure a plurality of valuesassociated with the input, the visual capturing device is configured tocapture a sequence of visual data associated with the plurality ofvalues, and the controller is configured to cause the plurality ofvalues and the sequence of visual data to be rendered on the displaydevice simultaneously and scroll through in time responsive to usermanipulation.

According to one embodiment, and various other features and advantageswill be apparent from a reading of the following detailed description.

BRIEF DESCRIPTION OF DRAWINGS

The embodiments are illustrated by way of examples, and not by way oflimitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements.

FIG. 1 shows an operating environment in accordance with someembodiments.

FIG. 2 shows components of a sensor based detection system in accordancewith some embodiments.

FIGS. 3A-3C show a sensor based system with playback in accordance withsome embodiments.

FIGS. 4A-4C show a sensor based system with playback graphical userinterface in accordance with some embodiments.

FIGS. 5A-5B show another sensor based playback graphical user interfacein accordance with some embodiments.

FIG. 6 shows a sensor based playback system and a display capturingaccording to some embodiments.

FIGS. 7A-7D show rendering of sensor readings and visual capturingdevice in a sensor based system in accordance with some embodiments.

FIGS. 8A-8C show rendering of another sensor readings and visualcapturing device in a sensor based system in accordance with someembodiments.

FIGS. 9A-9D show selection and display of sensors and their associatedvisual capturing devices in a sensor based system in accordance withsome embodiments.

FIGS. 10A-10B show data communication flow according to someembodiments.

FIG. 11A-11B shows a flow diagram according to some embodiments.

FIG. 12 shows a computer system in accordance with some embodiments.

FIG. 13 shows a block diagram of another computer system in accordancewith some embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to various embodiments, examples ofwhich are illustrated in the accompanying drawings. While the claimedembodiments will be described in conjunction with various embodiments,it will be understood that these various embodiments are not intended tolimit the scope. On the contrary, the claimed embodiments are intendedto cover alternatives, modifications, and equivalents, which may beincluded within the scope of the appended Claims. Furthermore, in thefollowing detailed description, numerous specific details are set forthin order to provide a thorough understanding of the claimed embodiments.However, it will be evident to one of ordinary skill in the art that theclaimed embodiments may be practiced without these specific details. Inother instances, well known methods, procedures, components, andcircuits are not described in detail so that aspects of the claimedembodiments are not obscured.

Some portions of the detailed descriptions that follow are presented interms of procedures, logic blocks, processing, and other symbolicrepresentations of operations on data bits within a computer memory.These descriptions and representations are the means used by thoseskilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. In the presentapplication, a procedure, logic block, process, or the like, isconceived to be a self-consistent sequence of operations or steps orinstructions leading to a desired result. The operations or steps arethose utilizing physical manipulations of physical quantities. Usually,although not necessarily, these quantities take the form of electricalor magnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a computer system or computingdevice. It has proven convenient at times, principally for reasons ofcommon usage, to refer to these signals as transactions, bits, values,elements, symbols, characters, samples, pixels, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present disclosure,discussions utilizing terms such as “receiving,” “converting,”“transmitting,” “storing,” “determining,” “sending,” “querying,”“providing,” “accessing,” “associating,” “configuring,” “initiating,”“customizing”, “mapping,” “modifying,” “analyzing,” “displaying,” or thelike, refer to actions and processes of a computer system or similarelectronic computing device or processor. The computer system or similarelectronic computing device manipulates and transforms data representedas physical (electronic) quantities within the computer system memories,registers or other such information storage, transmission or displaydevices.

It is appreciated that present systems and methods can be implemented ina variety of architectures and configurations. For example, presentsystems and methods can be implemented as part of a distributedcomputing environment, a cloud computing environment, a client serverenvironment, etc. Embodiments described herein may be discussed in thegeneral context of computer-executable instructions residing on someform of computer-readable storage medium, such as program modules,executed by one or more computers, computing devices, or other devices.By way of example, and not limitation, computer-readable storage mediamay comprise computer storage media and communication media. Generally,program modules include routines, programs, objects, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. The functionality of the program modules may becombined or distributed as desired in various embodiments.

Computer storage media can include volatile and nonvolatile, removableand non-removable media implemented in any method or technology forstorage of information such as computer-readable instructions, datastructures, program modules, or other data, that are non-transitory.Computer storage media can include, but is not limited to, random accessmemory (RAM), read only memory (ROM), electrically erasable programmableROM (EEPROM), flash memory, or other memory technology, compact disk ROM(CD-ROM), digital versatile disks (DVDs) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storethe desired information and that can be accessed to retrieve thatinformation.

Communication media can embody computer-executable instructions, datastructures, program modules, or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media can include wired media such asa wired network or direct-wired connection, and wireless media such asacoustic, radio frequency (RF), infrared and other wireless media.Combinations of any of the above can also be included within the scopeof computer-readable storage media.

A need has arisen for a solution to allow monitoring and collection ofdata from a plurality of sensors and management of the plurality ofsensors for improving security of our communities, e.g., by detectingradiation, etc. Further, there is a need to provide relevant informationbased on the sensors in an efficient manner to increase security.According to some embodiments, one or more sensor readings or associatedderived data may be displayed in a graphical user interface (GUI) foruser manipulation. For example, the operator can scroll through, e.g.,rewind, forward, play, pause, etc., information, e.g., sensor readings,change of state of sensor(s), condition being met by one or moresensors, etc., over a time period of interest. It is appreciated thatthe time period of interest may be user selectable, e.g., from a weekago to present time including real-time data, etc. As such, occurrenceof a possible event, e.g., possible terrorist activity, possibleradiation from a bio-hazardous material, etc., may be detected usingplayback functionality of the GUI, thereby circumventing a catastrophicevent.

Furthermore, a need has arisen to complement the sensor based detectionsystem with surveillance footage. According to some embodiments,surveillance footage may provide additional information regardingsensor(s) readings that are within the same physical proximity, e.g.,within the line of sight, within the same room, within the samebuilding, outside of the building monitoring the entrance to thebuilding, etc. As a result, the sensor based detection system equippedwith a surveillance system results in a preventative system as well as apunitive one.

According to some embodiments, a system may include a sensor, a visualcapturing device, and a controller. The sensor, e.g., thermal sensor,electromagnetic sensor, mechanical sensor, motion sensor,biological/chemical sensor, Geiger counter etc., may be configured tomeasure a value, e.g., thermal value, electromagnetic value, radiationvalue, etc., associated with an input, according to some embodiments.The visual capturing device, e.g., video camera, digital camera, etc.,may be configured to capture visual data e.g., video frames, pictures,etc. In some embodiments, the controller may be configured to cause themeasured value and the captured visual data to be rendered on a displaydevice simultaneously.

Embodiments provide methods and systems for monitoring and managing avariety of network (e.g., internet protocol (IP)) connected sensors.Embodiments are configured to allow monitoring (e.g., continuousreal-time monitoring, sporadic monitoring, scheduled monitoring, etc.)of sensors and associated sensor readings or data (e.g., ambient sensorreadings). For example, gamma radiation levels may be monitored in thecontext of background radiation levels. Accordingly, a significantchange in the background gamma radiation levels may indicate a presenceof hazardous radioactive material, (e.g., bomb, etc.). As a result,appropriate actions may be taken to avert a possible security breach,terrorist activity, etc. Embodiments may support any number of sensorsand may be scaled upwards or downwards as desired. Embodiments thusprovide a universal sensor monitoring, management, and alertingplatform.

Embodiments provide analytics, archiving, status (e.g., real-timestatus, sporadic monitoring, scheduled monitoring, etc.), GUI basedmonitoring and management to enable one to scroll through sensor(s)reading and/or captured surveillance footage over time. In someembodiments, the system may include a messaging system to alert thecommunity regarding certain risks. Embodiments may provide a solutionfor monitoring, managing, alerting, and messaging related to certainsensor detection, e.g., gamma radiation detection, air qualitydetection, water and level quality detection, fire detection, flooddetection, biological and chemical detection, air pressure detection,particle count detection, movement and vibration detection, etc. Forexample, the embodiments may provide a solution for monitoring andtracking movement of hazardous materials or conditions, thereby allowinginitiation of public responses and defense mechanisms. Embodiments mayallow previously installed devices (e.g., surveillance cameras,smartphones, vibration detection sensors, carbon monoxide detectionsensors, particle detection sensors, air pressure detection sensors,infrared detection sensors, etc.) to be used as sensors to detecthazardous conditions (e.g., radioactive, biological, chemical, etc.).

Embodiments may be used in a variety of environments, including publicplaces or venues (e.g., airports, bus terminals, stadiums, concerthalls, tourist attractions, public transit systems, etc.), organizations(e.g., businesses, hospitals, freight yards, government offices, defenseestablishments, nuclear establishments, laboratories, etc.), etc. Forexample, embodiments may be used to track sensitive material (e.g.,nuclear, biological, chemical, etc.) to ensure that it is not releasedto the public and prevent introduction of the material into publicareas. Embodiments may thus be further able to facilitate a rapidresponse to terrorist threats (e.g., a dirty bomb). It is appreciatedthat the embodiments described herein are within the context ofradiation detection and gamma ray detection for merely illustrativepurposes and are not intended to limit the scope.

FIG. 1 shows an operating environment in accordance with someembodiments. The operating environment 100 includes a sensor baseddetection system 102, a network 104, a network 106, a messaging system108, sensors 110-120, and visual capturing device(s) 130. The sensorbased detection system 102 and the messaging system 108 are coupled to anetwork 104. The sensor based detection system 102 and messaging system108 are communicatively coupled via the network 104. The sensor baseddetection system 102, sensors 110-120, and visual capturing device(s)130 are coupled to a network 106. The sensor based detection system 102,sensors 110-120, and visually capturing device(s) 130 arecommunicatively coupled via network 106. Networks 104, 106 may includemore than one network (e.g., intranets, the Internet, local areanetworks (LAN)s, wide area networks (WAN)s, wireless local area network(WiFi), etc.) and may be a combination of one or more networks includingthe Internet. In some embodiments, network 104 and network 106 may be asingle network.

The sensors 110-120 detect a reading associated therewith, e.g., gammaradiation, vibration, heat, motion, etc., and transmit that informationto the sensor based detection system 102 for analysis. The sensor baseddetection system 102 may use the received information and compare it toa threshold value, e.g., historical values, user selected values, etc.,in order to determine whether a potentially hazardous event hasoccurred. In response to the determination, the sensor based detectionsystem 102 may transmit that information to the messaging system 108 forappropriate action, e.g., emailing the appropriate personnel, soundingan alarm, tweeting an alert, alerting the police department, alertingthe homeland security department, etc. Accordingly, appropriate actionsmay be taken in order to respond to the risk.

It is appreciated that the sensor based detection system 102 may providethe sensor readings or any information derived therefrom in a GUI. Forexample, readings from various sensors may be displayed over time suchthat their historical readings may be compared to their currentreadings, etc. In some embodiments, derived information from the sensorreadings may be rendered on the GUI. For example, the state of thesensor may be displayed, e.g., whether the sensor is in normal mode,elevated mode, highly elevated mode, etc., over time. In other words,the GUI may enable a user to scroll through sensor readings or derivedinformation therefrom over time, e.g., the time may be user selected.

According to some embodiments, the sensor based detection system 102 mayreceive visual data and queues from the visual capturing devices 130. Insome embodiments, the visual capturing device(s) 130 are associated withone or more of the sensors 110-120. Data captured, e.g., stilledfootage, video frame, infrared information, etc., from the visualcapturing device(s) 130 may complement information received from thesensors 110-120. For example, sensor readings or derived informationfrom the sensors may be displayed on a GUI as well as the visual datathat was captured, thereby making it easier to detect a possiblehazardous situation. As an illustrative example, an elevated readingfrom one or more sensors 110-120 may cause one to inspect footage fromthe visual capturing device(s) 130, therefore identifying a leak in apipe, identifying a possible terrorist setting up a bomb, etc.

The sensors 110-120 may be any of a variety of sensors including thermalsensors (e.g., temperature, heat, etc.), electromagnetic sensors (e.g.,metal detectors, light sensors, particle sensors, Geiger counter,charge-coupled device (CCD), etc.), mechanical sensors (e.g. tachometer,odometer, etc.), complementary metal-oxide-semiconductor (CMOS),biological/chemical (e.g., toxins, nutrients, etc.), etc. The sensors110-120 may further be any of a variety of sensors or a combinationthereof including, but not limited to, acoustic, sound, vibration,automotive/transportation, chemical, electrical, magnetic, radio,environmental, weather, moisture, humidity, flow, fluid velocity,ionizing, atomic, subatomic, navigational, position, angle,displacement, distance, speed, acceleration, optical, light imaging,photon, pressure, force, density, level, thermal, heat, temperature,proximity, presence, radiation, Geiger counter, crystal based portalsensors, biochemical, pressure, air quality, water quality, fire, flood,intrusion detection, motion detection, particle count, water level,surveillance cameras, etc. The sensors 110-120 may be video cameras(e.g., IP video cameras) or purpose built sensors.

The sensors 110-120 and visual capturing device(s) 130 may be fixed inlocation (e.g., surveillance cameras or sensors), semi-fixed (e.g.,sensors on a cell tower on wheels or affixed to another semi portableobject), mobile (e.g., part of a mobile device, smartphone, etc.), orany combination thereof. The sensors 110-120 may provide data to thesensor based detection system 102 according to the type of the sensors110-120. For example, sensors 110-120 may be CMOS sensors configured forgamma radiation detection. Gamma radiation may thus illuminate a pixel,which is converted into an electrical signal and sent to the sensorbased detection system 102.

The sensor based detection system 102 is configured to receive data andmanage sensors 110-120. The sensor based detection system 102 isconfigured to assist users in monitoring and tracking sensor readings orlevels at one or more locations. The sensor based detection system 102may have various components that allow for easy deployment of newsensors within a location (e.g., by an administrator) and allow formonitoring of the sensors to detect events based on user preferences,heuristics, etc. The events may be used by the messaging system 108 togenerate sensor-based alerts (e.g., based on sensor readings above athreshold for one sensor, based on the sensor readings of two sensorswithin a certain proximity being above a threshold, etc.) in order forthe appropriate personnel to take action. The sensor based detectionsystem 102 may receive data and manage any number of sensors, which maybe located at geographically disparate locations. In some embodiments,the sensors 110-120 and components of a sensor based detection system102 may be distributed over multiple systems (e.g., and virtualized) anda large geographical area.

It is appreciated that the information received from one or moresensors, e.g., sensors 110-120, may be complemented with data capturedusing the visually capturing device(s) 130. For example, visuallycaptured data from one or more video camera, still camera, infraredcamera, etc., may also be gathered and transmitted to the sensor baseddetection system 102 for processing. Accordingly, detection of apossible hazardous situation becomes easier, e.g., leak from a pipe,possible terrorist transporting bio-hazardous material, etc. As such,circumventing the possible hazardous situation becomes easier and theinformation may be used, for example, in adjudication of a terroristattack. It is appreciated that in some embodiments visual capturing ofdata is initiated in response to one or more sensors satisfying acertain condition, e.g., sensor readings above a certain threshold. Insome embodiments, the visually captured data may be time stamped andstored for later retrieval.

The sensor based detection system 102 may track and store locationinformation (e.g., board room B, floor 2, terminal A, etc.) and globalpositioning system (GPS) coordinates, e.g., latitude, longitude, etc.for each sensor, or group of sensors as well as for the visual capturingdevice(s) 130. The sensor based detection system 102 may be configuredto monitor sensors and track sensor values to determine whether adefined event has occurred, e.g., whether a detected radiation level isabove a certain threshold, etc. It is appreciated that the sensor baseddetection system 102 may be configured to also monitor, track and storedata associated with visual capturing device(s) 130. It is appreciatedthat in some embodiments, data may be captured by the visual capturingdevice(s) 130 when the sensor(s) satisfies a certain condition, e.g.,sensor(s) readings at a certain threshold, above a certain threshold,etc. However, in other embodiments data from the visual capturingdevices 130 may occur in absence of a certain condition being satisfiedby the sensor(s).

The sensor based detection system 102 may determine a route or path oftravel that dangerous or contraband material is taking around or withinrange of the sensors. For example, the path of travel of radioactivematerial relative to fixed sensors may be determined and displayed via aGUI. It is appreciated that the path of travel of radioactive materialrelative to mobile sensors, e.g., smartphones, etc., or relative to amixture of fixed and mobile sensors may similarly be determined anddisplayed via a GUI. It is appreciated that the analysis and/or thesensed values may be displayed in real-time or stored for laterretrieval. The GUI enables the user to scroll through events in sequenceby playing, fast forwarding, rewinding, pausing, etc.

The visual capturing device(s) 130 may be controlled by the sensor baseddetection system 102. For example, after the sensor based detectionsystem 102 determines a route or path of travel, the visual capturingdevice(s) may be reoriented and/or refocused to capture the path oftravel in real-time. In other words, the visual capturing device(s) maymove, sweep, or become oriented based on measured information by thesensor(s) and as determined by the sensor based detection system 102.The visual capturing device(s) may respond as sensors measure values andas the sensors are activated and de-activated. Groups of visualcapturing devices may also be employed to maximize coverage of theroute, or predicted path of travel. For example, the visual capturingdevice nearest the activated sensor may focus on the precise location ofthe sensor, while other visual capturing devices in the area may panacross the room/area to increase coverage.

The sensor based detection system 102 may display a GUI for monitoringand managing sensors 110-120 as well as the visual capturing device(s)130. The GUI may be configured for indicating sensor readings, sensorstatus, sensor locations on a map, etc., as well as rendering captureddata by the visual capturing device(s) 130, their respective locationson a map, etc. The sensor based detection system 102 may allow review ofpast sensor readings and movement of sensor detected material orconditions based on stop, play, pause, fast forward, and rewindfunctionality of stored sensor values as well as the captured visualdata by the visual capturing device(s) 130. The sensor based detectionsystem 102 therefore allows viewing of image or video footagecorresponding to sensors that had sensor readings above a threshold(e.g., based on a predetermined value or based on ambient sensorreadings). For example, a sensor may be selected in a GUI and videofootage associated with an area within a sensor's range of detection maybe displayed, thereby enabling a user to see an individual transportinghazardous material. According to one embodiment the footage is displayedin response to a user selection or it may be displayed automatically inresponse to a certain event, e.g., sensor reading associated with aparticular sensor or group of sensors being above a certain threshold.

In some embodiments, sensor readings of one or more sensors may bedisplayed on a graph or chart for easy viewing. A visual map-baseddisplay depicting sensors may be displayed with the sensors color codedaccording to the sensors' readings and certain events. For example, graymay be associated with a calibrating sensor, green may be associatedwith a normal reading from the sensor, yellow may be associated with anelevated sensor reading, orange associated with a potential hazardsensor reading, and red associated with a hazard alert sensor reading.It is appreciated that a representation of the visual capturingdevice(s) 130 may similarly be displayed in the GUI similar to thesensors (as presented above). Captured data by the visual capturingdevice(s) 130 may also be rendered on the GUI.

The sensor based detection system 102 may determine alerts or sensorreadings above a specified threshold (e.g., predetermined, dynamic, orambient based) or based on heuristics and display the alerts in the GUI.The sensor based detection system 102 may allow a user (e.g., operator)to group multiple sensors together to create an event associated withmultiple alerts from multiple sensors. For example, a code red event maybe created when three sensors or more within twenty feet of one anotherand within the same physical space have a sensor reading that is atleast 40% above the historical values. In some embodiments, the sensorbased detection system 102 may automatically group sensors togetherbased on geographical proximity of the sensors, e.g., sensors of gates1, 2, and 3 within terminal A at LAX airport may be grouped together dueto their proximate location with respect to one another, e.g., physicalproximity within the same physical space, whereas sensors in differentterminals may not be grouped because of their disparate locations.However, in certain circumstances sensors within the same airport may begrouped together in order to monitor events at the airport and not at amore granular level of terminals, gates, etc. In other words, sensorsmay be grouped and scaled as desired.

It is appreciated that visual capturing device(s) 130 may similarly begrouped, managed, and captured information therefrom may be stored forlater retrieval or it may be rendered without storing it. According tosome embodiments, the visual capturing devices 130 may be grouped withone another and/or with sensors 110-120.

The sensor based detection system 102 may send information to amessaging system 108 based on the determination of an event created fromthe information collected from the sensors 110-120 and/or visualcapturing device(s) 130. The messaging system 108 may include one ormore messaging systems or platforms which may include a database (e.g.,messaging, SQL, or other database), short message service (SMS),multimedia messaging service (MMS), instant messaging services, Twitter™available from Twitter, Inc. of San Francisco, Calif., Extensible MarkupLanguage (XML) based messaging service (e.g., for communication with aFusion center), JavaScript™ Object Notation (JSON) messaging service,etc. For example, national information exchange model (NIEM) compliantmessaging may be used to report chemical, biological, radiological andnuclear defense (CBRN) suspicious activity reports (SARs) to report togovernment entities (e.g., local, state, or federal government).

FIG. 2 shows components of a sensor based detection system in accordancewith some embodiments. Diagram 200 includes sensors 250-260, a network230, a sensor based detection system 202, and a visual capturingdevice(s) 205. The sensor based detection system 202, sensors 250-260,and the visual capturing device(s) 205 are communicatively coupled vianetwork 230. The network 230 may include more than one network (e.g.,intranets, the Internet, LANs, WANs, etc.) and may be a combination ofone or more networks including the Internet. In some embodiments, thesensors 250-260 may be substantially similar to sensors 110-120 and maybe any of a variety of sensors, as described herein. It is appreciatedthat the visual capturing device(s) 205 may be substantially similar tothe visual capturing device(s) 130 and it may operate substantially insimilar fashion.

The sensor based detection system 202 may access or receive data fromthe sensors 250-260. It is appreciated that the detection system 202 mayalso receive surveillance footage from visual capturing device(s) 205.According to some embodiments, the surveillance footage is receivedregardless of the status of the sensors, however, in some embodimentsthe surveillance footage is received when one or more sensors satisfy acertain condition, e.g., detected measurement exceeding a certainthreshold, a majority of the sensors registering a certain change instatus, etc.

The sensor based detection system 202 may include a sensor managementmodule 204, a sensor process module 206, a data warehouse module 208, astate management module 210, a visualization module 212, a messagingmodule 214, a location module 216, and a user management module 218.

In some embodiments, the sensor based detection system 202 may bedistributed over multiple servers (e.g., physical or virtual machines).For example, a domain server may execute the data warehouse module 208and the visualization module 212, a location server may execute thesensor management module 204 and one or more instances of a sensorprocess module 206, and a messaging server may execute the messagingmodule 214. For example, multiple location servers may each be locatedat respective sites having 100 sensors, and provide analytics to asingle domain server, which provides a monitoring and managementinterface (e.g., GUI) and messaging services. The domain server may becentrally located while the location servers may be located proximate tothe sensors for bandwidth purposes.

The sensor management module 204 is configured to monitor and manage thesensors 250-260. The sensor management module 204 is configured toinitiate one or more instances of sensor process module 206 formonitoring and managing sensors 250-260. The sensor management module204 is operable to configure a new sensor process (e.g., an instance ofsensor process module 206) when a new sensor is installed. The sensormanagement module 204 may thus initiate execution of multiple instancesof the sensor process module 206. In some embodiments, an instance ofthe sensor process module 206 is executed for each sensor. For example,if there are 50 sensors, 50 instances of sensor process module 206 areexecuted in order to configure the sensors.

It is further appreciated that the sensor management module 204 may alsobe operable to configure an already existing sensor. For example, sensor252 may have been configured previously, however, the sensor managementmodule 204 may reconfigure sensor 252 based on the new configurationparameters. The sensor management module 204 may be configured as anaggregator and collector of data from the sensors 250-260 via sensorprocess module 206. Sensor management module 204 is configured to senddata received via instances of sensor process module 206 to a datawarehouse module 208.

The sensor management module 204 further allows monitoring of one ormore instances of the sensor process module 206 to determine whether aninstance of the sensor process module 206 is running properly or not. Insome embodiments, the sensor management module 204 is configured todetermine the health of one or more sensors including if a sensor hasfailed based on, for example, whether an anticipated or predicted valueis received within a certain time period. The sensor management module204 may further be configured to determine whether data is arriving ontime and whether the data indicates that the sensor is functioningproperly (e.g. healthy) or not. For example, a radiation sensor may beexpected to provide a certain microsievert (mSv) value within a giventime period. In some embodiments, the anticipated value may be receivedfrom an analytics engine that analyzes the sensor data. In someembodiments, the sensor management module 204 may be configured toreceive an indicator of status from a sensor (e.g., an alive signal, anerror signal, or an on/off signal). The health information may be usedfor management of the sensors 250-260 and the health informationassociated with the sensors may be stored in the data warehouse 208.

The sensor management module 204 may further access and examine theoutputs from the sensors based on a predictable rate of output. Forexample, an analytics process (e.g., performed by the sensor processmodule 206) associated with a sensor may produce a record every tenseconds and if a record is not received (e.g., within multiple 10 secondperiods of time), the sensor management module 204 may stop and restartthe analytics process. In some embodiments, the record may be a flatfile.

The sensor process module 206 is configured to receive data (e.g., bulkor raw data) from sensors 250-260. In some embodiments, the sensorprocess module 206 may form a record (e.g. a flat file) based on thedata received from the sensors 250-260. The sensor process module 206may perform analysis of the raw data (e.g., analyze frames of video todetermine sensor readings). In some embodiments, the sensor processmodule 206 may then pass the records to the sensor management module204.

The data warehouse module 208 is configured to receive data from sensormanagement module 204. The data warehouse module 208 is configured forstoring sensor readings and metadata associated with the sensors.Metadata for the sensors may include their respective geographicalinformation (e.g., GPS coordinates, latitude, longitude, etc.),description of the sensor and its location, e.g., sensor at gate 1terminal A at LAX, etc. In some embodiments, the data warehouse module208 may be configured to determine state changes based on monitoring(e.g., real-time monitoring) of the state of each sensor and the stateof the sensor over a time interval (e.g., 30 seconds, 1 minute, 1 hour,etc.). In some embodiments, the data warehouse module 208 is configuredto generate an alert (e.g., when a sensor state has changed and is abovea threshold, when a sensor reading satisfies a certain condition such asbeing below a threshold, etc.). The generated alert may be sent tovisualization module 212 for display (e.g., to a user). Changes insensor state may thus be brought to the attention of a user (e.g.,operator). It is appreciated that the threshold values may be one ormore historical values, safe readings, operator selected values, etc.

It is appreciated that other information may similarly be displayed viathe visualization module 212. For example, actual reading of the sensormay be displayed on the GUI that displays a geographical map and thesensor located therein. It is appreciated that additional informationmay be displayed in response to a user selection, e.g., GPS coordinatesof the sensor may be displayed when the user selects the displayedsensor, the type and brand of the sensor may be displayed when the userselects the displayed sensor, etc. It is appreciated that thevisualization module 212 displaying the GUI with the sensor readings mayalso display a playback functionality for sensors such that the user canscroll back and forth in time, play, pause, etc., to see measurements bythe sensors in different time intervals. It is also appreciated thataccording to some embodiments, the playback functionality may becomplemented with the surveillance footage captured by the visualcapturing device(s) 205, similar to FIG. 1.

In some embodiments, the data warehouse module 208 may be implemented ina substantially similar manner as described in Philippines PatentApplication No. 1-2013-000136 entitled “A Domain Agnostic Method andSystem for the Capture, Storage, and Analysis of Sensor Reading”, byFerdinand E. K. De Antoni (Attorney Docket No. 13-027-00-PH) which isincorporated by reference herein.

The state management module 210 may read data from the data warehousemodule 208 and/or from the sensor management module 204 (e.g., data thatwas written by sensor management module 204) and determine whether astate change has occurred. The state change may be determined based on aformula to determine whether there has been a change since a previousrecord in time for an associated sensor and may take into accountambient sensor readings. If there is a change in state, an alert may betriggered. It is appreciated that state may also be a range of values.One or more alerts may be assembled (e.g., into a data structure)referred to as an event. The event may then be accessed by or sent to avisualization module 212. The visualization module 212 may then displaythe change in state, an alert, or an event. In some embodiments, thevisualization module 212 may receive input to have the alert sent to anexternal system (e.g., a messaging system). It is appreciated that thevisualization module 212 may display a playback functionality forrendered information, e.g., the state, alert, events, measurementreadings from sensors, etc. As such, the operator may scroll throughvalues, states, events, alerts, etc., over any time interval ofinterest.

The visualization module 212 is configured for use in monitoring alocation for potential sensor based alerts. The visualization module 212may provide a GUI to monitor and manage each of the deployed sensors. Insome embodiments, the visualization module 212 is configured to providea tree filter to view each of the sensors in a hierarchical manner, aswell as a map view, thereby allowing monitoring of each sensor in ageographical context. The visualization module 212 may further allowcreation of an event case file to capture sensor alerts at any point intime and escalate the sensor alert to appropriate authorities forfurther analysis (e.g., via a messaging system). The visualizationmodule 212 may display a path of travel or route of hazardous materialsor conditions based on sensor readings and the associated sensorlocations. The visualization module 212 may further be used to zoom inand zoom out on a group of sensors, e.g., sensors within a terminal atan airport, etc. As such, the information may be displayed as granularas desired by the operator. Visualization module 212 may also be usedand render information in response to a user manipulation. For example,in response to a user selection of a sensor, e.g., sensor 260, thesensor readings associated with the sensor may be displayed. In anotherexample, a video feed associated with the sensor may also be displayed(e.g., simultaneously).

The messaging module 214 is configured to send messages to other systemsor messaging services including, but not limited to, a database (e.g.,messaging, SQL, or other database), short message service (SMS),multimedia messaging service (MMS), instant messaging services, Twitteravailable from Twitter, Inc. of San Francisco, Calif., Extensible MarkupLanguage (XML) based messaging service (e.g., for communication with aFusion center), JavaScript Object Notation (JSON) messaging service,etc. In one example, national information exchange model (NIEM)compliant messaging may be used to report chemical, biological,radiological and nuclear defense (CBRN) suspicious activity reports(SARs) to report to government entities (e.g., local, state, or federalgovernment). In some embodiments, the messaging module 214 may sendmessages based on data received from the sensor management module 204.It is appreciated that the messages may be formatted to comply with therequirement/standards of the messaging service used. For example, asdescribed above a message may be formed into the NIEM format in order torepot a CBRN event.

The location module 216 is configured for mapping and spatial analysis(e.g., triangulation) in order to graphically represent the sensorswithin a location. For example, location module 216 may be configured tofacilitate display of the location of and associated icons for sensorsat each gate of an airport terminal In some embodiments, the sensormanagement module 204 is configured to store geographical dataassociated with a sensor in a data store (not shown) associated withlocation module 216. In some embodiments, the location module 216 mayoperate in conjunction with ArcGIS from ERSI, Inc. of Redlands, Calif.It is appreciated that the location module 216 may be used to providemapping information associated with the sensor location such that thelocation of the sensor may overlay the map, e.g., location of the sensormay overlay the map of LAX airport, etc.

The user management module 218 is configured for user management andstorage of user identifiers of operators and administrators. The usermanagement portion may be integrated with an existing user managementsystems (e.g., OpenLDAP or Active Director) thereby enabling use ofexisting user accounts to operate sensor the based detection system 202.

Referring now to FIGS. 3A-3C, a sensor based system with playback inaccordance with some embodiments is shown. Referring more specificallyto FIG. 3A, system 300 is shown. System 300 includes sensors 310 a, 310b, 310 c, 310 d, 310 e, a storage 370 component, a controller 340, and adisplay device 380.

According to some embodiments, the sensors 310 a-310 e measure valuesassociated with their respective input(s). Sensors 310 a-310 e may beany combination of sensors described in FIG. 1 and may measure anycombination of values associated with their respective inputs, asdescribed in FIG. 1. The measured information by sensors 310 a-310 e maybe transmitted to the controller 340 for processing. The processedinformation and/or raw data received from the sensors may be stored inthe storage 370 component for later retrieval. Raw information,processed information, sensor related information, or any combinationthereof may be rendered on the display device 380 as a GUI.

The controller 340 may process the received information e.g., measuredvalues, information associated with the sensors, etc., to generatederived information, e.g., state of sensor (e.g., normal state, elevatedstate, critical state, calibration state, etc.), change in state (e.g.,change from normal to elevated, change from critical to elevated, etc.),etc. In some embodiments, the measured information by the sensors 310a-310 e may be stored in the storage 370 component, either directly (notshown) or through the controller 340. It is appreciated that accordingto some embodiments, the controller 340 may cause the generated derivedinformation to be stored in the storage 370 component. In someembodiments, the controller 340 may render the received information orany information derived therefrom to be displayed on the display device380 as a GUI. It is further appreciated that the controller 340 mayrender information associated with the sensors, e.g., name of thesensor, sensor type, geo-locational position of the sensor, metadataassociated with the sensor, etc., as a GUI on the display device 380.

In some embodiments, the measured data by the sensors 310 a-310 e,and/or the derived data from the measured data may be time stamped. Timestamping of the measured data and/or the derived data therefrom mayoccur automatically, e.g., time stamping every measured value, timestamping every other measured value, time stamping in a certaininterval, etc. In some embodiments, time stamping the measured dataand/or the derived data therefrom may be based on satisfaction ofcertain conditions, e.g., the sensor reading exceeding a threshold, agroup of sensor readings exceeding a threshold within a certain period,etc. In some embodiment, time stamping the measured data and/or deriveddata therefrom may be responsive to a user selection to time stamp.

The display device 380 may render information, e.g., measured values,derived information, etc., associated with any combination of sensors,e.g., sensors 310 a-310 d but not 310 e. The measured values and/orderived information may include historical and/or real-time values. FIG.3A shows rendering of measured values for sensors 310 a-310 d forillustrative purposes but it is not limited thereto, e.g., derivedinformation may be rendered. Display device 380 renders the sensorsreadings, and the threshold value of interest, e.g., temperature ofinterest, vibration of interest, radiation of interest, etc., over time.Accordingly, the user can identify the time which any given sensor has areading above or below the threshold value of interest. It isappreciated that rendition of the threshold value is for illustrativepurposes only and should not construed as limiting the scope of theembodiments. It is also appreciated that throughout the applicationrendition of sensor reading is discussed for illustrative purposes butthe embodiments should not be construed as limiting the scope. Forexample, the sensor reading may include derived information, etc., asdiscussed above.

It is appreciated that the time interval may be user adjusted, e.g., theuser may select to display measured values for the past 1 hour, the usermay select to display measured values between 3 pm to 6 pm on Sep. 16,2008, etc. According to some embodiments, the time interval of interestmay be selected using a pop-up window, drop-down menu, etc.

According to some embodiments, the display device 380 may also render aplayback device 382. The playback device 382 enables the user to scrollthrough the time interval of interest in sequence. The playback device382 may include various functionalities, e.g., play, stop, pause (notshown), fast forward, rewind, next chapter, previous chapter, etc. Inother words, the user may see changes in measured values for each sensorfor the time interval of interest. In this illustrative embodiment, theplayback device 382 is associated with an event at LAX for sensors 310a-310 d over a time interval of interest.

The playback device 382 can be used to provide further insight and mayfurther help in drawing certain conclusions. For example, the playbackdevice 382 may reveal elevated readings and spike by sensors 310 a-310 din that chronological order over a short amount of time. Knowing thatsensors 310 a-310 d are within a certain physical proximity of oneanother and form a daisy chain topology, e.g., sensor 310 a close tosensor 310 b close to sensor 310 c close to sensor 310 d, and if thespike in measured values are in the same sequence may lead one to inferthat an event occurring between sensors 310 a-310 d caused the spike inreading, e.g., bio-hazardous material was being transported fromphysical proximity of sensor 310 a to 310 d. It is appreciated that theexample provided above is merely for illustrative purposes on how theplayback device 382 may be used and should not be construed as limitingthe embodiments. For example, the playback device 382 may be used tocreate an event, e.g., create conditions that when satisfied trigger analarm, based on historical analysis of the sensors.

Referring now to FIG. 3B, rendition of measured value, e.g., reading X1,for sensor 310 d is shown. According to some embodiments, the user mayselect, e.g., by clicking, the sensor reading of interest at a giventime of interest to show the measured value at the time of interest. Insome embodiments, the user may cause the sensor reading at the giventime of interest to be displayed by hovering the pointing device, e.g.,mouse, over the sensor reading at given time of interest.

Referring now to FIG. 3C, measured values at a given time associatedwith each sensor, e.g., sensors 310 a-310 d, may be displayed as theuser manipulated the moving window 381 of the GUI. For example,manipulating the moving window 381 to the left or to the right enablesthe user to see measured values X1-X4 associated with sensors 310 a-310d respectively. It is appreciated that in some embodiments, the movingwindow 381 may include two vertical lines (not shown) in order to alsoselect the time interval of interest.

Referring now to FIGS. 4A-4C, a sensor based system with playbackgraphical user interface in accordance with some embodiments is shown.Referring to FIG. 4A, the display device 380 may render a map pane 404and location pane 406. The map pane 404 may display the map associatedwith sensors. In this illustrative embodiment, the map of California isdisplayed with 24 sensors located in Southern California. It isappreciated that information displayed on the map may be zoomed in andzoomed out as desired using various tools, e.g., zooming device 402. Inresponse to zooming in/out the sensors may be collapsed into groups,e.g., based on geographical location, based on type, etc., and expandedout accordingly.

It is appreciated that rendition on the map pane 404 may be controlledby manipulating the location pane 406. The location pane 406 may includeavailable locations for the sensors, search box for finding sensors ofinterest, and it may further display the locational information inhierarchical format, e.g., LAX Terminal 1 gate 11 having 3 sensors, LAXTerminal 1 gate 12 having 4 sensors, etc. The location pane 406 mayfurther include saved locations of interest with its own search tool.

Referring now to FIG. 4B, zooming in on some of the 24 sensors is shown.For example, once the user further zooms in using the zooming tool 402,seven sensors of interest, e.g., located at gate 11 and 12 of LAXterminal 1, out of the twenty four sensors are displayed. It isappreciated that the zooming in/out may also be achieved by manipulatingthe location pane 406 window, e.g., by selecting LAX terminal 1 allsensors in terminal 1 would be displayed in the map pane 404. It isappreciated that in this embodiment, the user may be interested insensors of gate 11. As such, gate 11 may be selected by selecting it inthe location pane 406 and/or by selecting it in the map pane 404. It isappreciated that user selection of gate 11 may be through any othermeans, e.g., hovering the pointing device over the group of sensors ofinterest.

The selection of the gate of interest, e.g., gate 11, may result in theplayback device 382 to be displayed rendering the sensor readings overthe time of interest. In this embodiment, measured values from sensorsS1-S3 are displayed. The playback device 382 may be used in a similarfashion as described in FIGS. 3A-3C. For example, the playback device382 may be used in order to scroll through the measured values overtime, e.g., play, fast forward, rewind, etc.

Referring now to FIG. 4C, further zooming in on sensors of gate 11 maydisplay the geo-locational positions of sensors S1-S3 with respect toone another on the map in the map pane 404. It is appreciated thatzooming in/out may be through the zooming device 402 or via the locationpane 406, among other options. In this embodiment, selection of thesensor of interest, e.g., S3, via the map pane 404 or the location pane406 may cause the playback device 382 to render sensor readings for theselected sensor. In other words, while in FIG. 4B all three sensors ofgate 11 were displayed in the playback device 382, in FIG. 4C, userselection of sensor S3 may cause the playback device 382 to rendersensor readings associated with the selected sensor. It is appreciatedthat sensor selection may be through selection of the sensor in the mappane 404 by selecting or hovering over S3, through the location pane 406by selecting sensor 3 in gate 11, or by selecting the graph of sensorreadings associated with sensor 3 from the playback device 382 in FIG.4B, to name a few.

It is appreciated that selection of a given sensor may provideadditional information associated with the selected sensor. For example,selecting sensor 3 in gate 11 may provide information associated withthat sensor, e.g., configuration state of the sensor, last reset,duration of the state (e.g., duration of elevated state, duration ofcritical state, duration of normal state, etc.), name of the sensor,sensor type (e.g., sensor types as discussed in FIG. 1), closest sensorwithin its proximity, closest visual capturing device, longitudinal andlatitudinal information of the sensor, GPS location of the sensor,metadata associated with the sensor, etc.

Referring now to FIGS. 5A-5B, another sensor based playback graphicaluser interface in accordance with some embodiments is shown. FIG. 5A issimilar to that of FIG. 4B. However, in FIG. 5A, the sensors of interestare at gate 11 and gate 12 at terminal 1 of LAX. In this illustrativeembodiment, the playback device 382 renders the sensor readings for eachgate over the time interval of interest. For example, S_(1,G11) is themeasured reading for sensor 1 at gate 11, S_(2,G11) is the measuredreading for sensor 2 at gate 11, S_(3,G11) is the measured reading forsensor 3 at gate 11, S_(1,G12) is the measured reading for sensor 1 atgate 12, and S_(2,G12) is the measured reading for sensor 2 at gate 12,that are displayed by the playback device 382. It is appreciated thatthe playback device 382 may be manipulated by the user in the similarfashion as described above to provide insight to the user that otherwisewould go unnoticed.

Referring now to FIG. 5B, a path travelled by a hazardous material maybe displayed based on the playback device 382 and the measured sensorreadings. In this illustrative embodiment, S_(1,G11) shows an elevatedreading at time T1. However, measured reading of S_(1,G11) decreases attime T2, however, at time T2, measured reading of S_(2,G11) shows anelevated reading. Measured readings at time T3 further decreases forS_(1,G11) and S_(2,G11) while increases for S_(3,G11). The measuredsensor readings for sensors of gate 11 all decrease over times T4-T7,while readings for sensors S_(1,G12) and S_(2,G12) increase over thattime before it drops off. As such, the path travelled by hazardousmaterial may be rendered for visual representation of the event that mayhave occurred. It is appreciated that the path travelled by hazardousmaterial, e.g., via radiation detection, is exemplary and not intendedto limit the scope of the embodiments. For example, similar process maybe employed to detect dangerous temperatures at a manufacturing facilityat different machinery of an assembly line that may imply a possibledefect or hazard associated with a given batch of manufacturing product.

Referring now to FIG. 6, a sensor based playback system and a displaycapturing according to some embodiments is shown. System 600 includessensors 610 a-610 c that operate substantially similar to that ofsensors 310 a-310 e, a controller 640 that operates substantiallysimilar to that of controller 340, a storage 670 that operatessubstantially similar to that of storage 370, a display device 680 thatoperates substantially similar to that of display device 380, and avisual capturing devices 650 and 660.

It is appreciated that the visual capturing device 650 and 660 may besimilar to the visual capturing device 205 described above. For example,the visual capturing device 650 and 660 may include a video camera, adigital camera, a surveillance camera, a smartphone, a still camera, aninfrared camera, etc. The visual capturing device 650 and 660 maycapture information that is complementary to the measured values of thesensors.

According to one embodiment, the captured visual information by thevisual capturing device 650 and 660 may be displayed. For example, inthe illustrative embodiment described herein, the display device 680 mayrender the captured information by the visual capturing device 650 thatis associated with sensor 610 a. It is appreciated that the associationof the visual capturing device with one or more sensors is discussed insubsequent figures.

In this illustrative embodiment, the captured surveillance footagedisplayed along with the measured sensor may enable one to conclude thata spike in the measured value may be due to the woman passing in closeproximity of the sensor 610 a. As such, possible terrorist attacks maybe circumvented or detected for punitive measures. It is appreciatedthat the playback device 382 may be used to scroll through events intime for both the sensor readings and for the captured visualinformation. For example, fast forwarding using the playback device 382fast forwards the measured readings for sensor and also fast forwardsthe surveillance footage associated therewith.

According to some embodiments, the captured visual informationassociated with the visual capturing devices 650 and 660 is time stampedautomatically. In some embodiments, the captured visual information bythe visual capturing devices 650 and 660 may be time stamped based on auser selection to time stamp the captured visual information. Accordingto some embodiments, the captured visual information may be time stampedbased on heuristics. For example, the captured visual information may betime stamped in response to the sensor(s) meeting a certain criteria,e.g., being above a threshold, being within a certain range, sensorshaving a similar increase/decrease in measurement within a certainamount of time from one another, sensors of different type satisfyingdifferent set of criteria within a certain amount of time, etc. The timestamped captured visual information may be subsequently stored in thestorage component 670 for later retrieval, e.g., for simultaneouslyrendering the surveillance footage along with rendition of sensor(s)reading on the playback device 382. In other words, certain conditions,e.g., sensor(s) readings satisfying a certain condition, may trigger thecaptured visual information to be time stamped and stored for futureuse.

Referring now to FIGS. 7A-7D, rendering of sensor readings and visualcapturing device in a sensor based system in accordance with someembodiments is shown. The display device 680 may render the playbackdevice 382 and various selectable sensors, e.g., sensors 610 a-610 c.Referring now to FIG. 7B, user selection of sensor 610 a is shown. Inthis illustrative embodiment, user selection of sensor 610 a causessensor readings associated with sensor 610 a to be displayed along withthe captured visual information associated with visual capturing device650. In this illustrative embodiment, the visual capturing device 650 isassociated with sensor 610 a based on heuristics.

The playback device 382 may optionally be rendered to enable the user toscroll through events, e.g., sensor 610 a readings and captured visualinformation by visual capturing device 650, in a certain time period.For example, the user may select to play the event by selecting a playbutton on the playback device 382. Furthermore, the user may select tofast forward, rewind, etc., by manipulating the playback device 382. Insome embodiments, an interactive moving window 781 may optionally berendered to enable the user to scroll through events in time, e.g., bymoving the window 781 to the left the user can scroll back in time andby moving the window 781 to the right the user can scroll forward intime, etc.

It is appreciated that the time period for rendering the sensor readingand the captured visual information may be user selected, e.g., user mayselect to display events (sensor readings and captured visualinformation) from the past one hour, past one month, past one year, froma year ago to six months ago, etc. It is appreciated that period of timemay be set by default based on heuristics in some embodiments. Selectionof the time period may be similar to that of FIG. 3A-4C described above.

Referring not to FIG. 7C, another embodiment by which a sensor can beselected is shown. In this illustrative embodiment, a dropdown menu ispresented for user selection thereof. In response to user selection of asensor, e.g., sensor 610 a, from the dropdown menu, the capturedinformation, e.g., sensor readings and captured visual information ofthe visual capturing device, are rendered on the display device 380 as aGUI, as shown in FIG. 7D. It is appreciated that in some embodiments,the playback device 382 may optionally be rendered to enable the user tomanipulate scrolling through events in time. In some embodiments, aninteractive moving window 781 may optionally be rendered to enable theuser to scroll through events in time, e.g., by moving the window 781 tothe left the user can scroll back in time and by moving the window 781to the right the user can scroll forward in time, etc.

In some illustrative embodiments, the sensor(s) may be associated withone or more visual capturing device(s) based on a user selection. Insome embodiments, the sensor(s) may be associated with one or morevisual capturing device(s) based on heuristics. For example, sensor(s)may be associated with a visual capturing device based on variouscriteria, e.g., being within a certain proximity, being within a samestructure such as a building, being within a same room, being within theline of sight of one another, being within a similar geo-locationalposition, being within a similar GPS coordinates, being within a certaindistance and/or radius of one another, being associated with a samestructure such as exterior and interior of the same building or device,sharing similar latitudinal position, sharing similar longitudinalposition, etc.

Referring now to FIGS. 8A-8C, rendering of another sensor(s) readingsand visual capturing device in a sensor based system in accordance withsome embodiments is shown. In one illustrative embodiment, sensor(s) andthe visual capturing device(s) may be selected by the user, theselection of which may be independent from one another. In other words,the user may group the sensor(s) and the visual capturing device(s) asdesired. Referring now to FIG. 8B, user selection of sensor 610 a isshown. In this embodiment, the selection of the sensor 610 a causes thereading of the sensor 610 a to be displayed for a certain time period,e.g., user selected time period, default period, etc., as describedabove. It is appreciated that the sensor readings may be displayed onthe GUI optionally with the playback device 382 and/or the interactivemoving window 781 for user manipulation thereof.

Referring now to FIG. 8C, user selection of the visual capturing device650 in accordance with some embodiments is shown. In this embodiment,the selection of the visual capturing device 650 causes the capturedvisual information to be rendered on the GUI. Similar to above, therendered captured visual information may be manipulated using theplayback device 382 and/or the interactive moving window 781.

Referring now to FIGS. 9A-9D, selection and display of sensors and theirassociated visual capturing devices in a sensor based system inaccordance with some embodiments is shown. Referring specifically toFIG. 9A, the display device 680 rendering a GUI to enable the user toselect sensor(s) and visual capturing device(s) independently, orautomatic selection of sensor(s) and visual capturing device(s)automatically is shown. In some embodiments, information associated withall sensor(s) and the visual capturing device(s) or a subset thereof maybe displayed. In this embodiment, information regarding location, name,media access control (MAC) address, description latitude, longitude, andIP address of sensor(s) and visual capturing device(s) may be displayed.

In this embodiment, the user may select to individually andindependently select sensor(s), e.g., sensor 610 a, and visual capturingdevice(s), e.g., visual capturing device 650. The result of the userselection may be displayed in FIG. 9B in a similar fashion as FIGS. 8Band 8C.

Referring now to FIG. 9C, the GUI of the display device 680 may renderthe sensor(s), e.g., sensors 610 a-640 c, and the visual capturingdevice(s), e.g., visual capturing devices 650-660, for user selectionthereof along with their representation at their appropriate location ona map. It is appreciated that the representation of the sensor(s) andthe visual capturing device(s) on the map may be based on their actualphysical location. In this embodiment, the map is the map of LAX airportwith sensor(s) and visual capturing device(s) positioned on the map thatrepresent the actual sensor(s) and visual capturing device(s).

According to some embodiments, the user may select the sensor(s) and thevisual capturing device(s) as desired. In other words, the user maygroup the sensors and the visual capturing devices as desired. In thisillustrative embodiment, the user has selected sensors 610 b and 610 calong with the visual capturing device 660. Thus, sensors 610 b-c andthe visual capturing device 660 may be grouped together and theirinformation may be rendered on the GUI, as shown in FIG. 9D. It isappreciated that the user may scroll through events, measurements, andtime by manipulating the playback device 382 and/or the interactivemoving window 781.

In this embodiment, displaying measurements by sensors 610 b and 610 calong with the captured visual information by the visual capturingdevice 660 may provide certain queues to the user. In other words, thevisual capturing device 660 may provide valuable visual queues tocomplement measurement information by the sensors. For example, the usermay observe that a female individual has been walking between sensors610 b and 610 c as visually captured by the visual capturing device 660.Combining the visual queues along with the sensor(s) readings mayprovide clues for elevated measurement readings for the sensors, e.g.,indicating that the female individual was the cause of elevated reading,therefore perhaps carrying bio-hazardous material. As such, appropriateaction may be taken, e.g., notifying the appropriate personnel such aspolice, locking down the facility, sending a tweet to public to reportseeing the female suspect, etc.

Referring now to FIGS. 10A-10B, data communication flow according tosome embodiments is shown. Referring specifically to FIG. 10A, sensor1010 transmits sensor data, e.g., measured from its input, to thecontroller 1040. The controller 1040 may process the received sensordata. In one embodiment, the controller 1040 may determine that thesensor data and/or data derived therefrom is to be stored in the storagecomponent 1070 and is transmitted for storage thereof. In someembodiments, the controller 1040 may determine whether a triggeringevent has occurred, e.g., certain measurement reading, certainmeasurement reading by a group of sensors, certain measurement readingby a group of sensors within a certain period of time, etc. Thecontroller 1040 may trigger a signal in response to determining that thetriggering event has occurred. The triggering event is transmitted tothe visual capturing device 1050, in one embodiment. As such, the visualcapturing device 1050 may transmit captured visual information to thecontroller 1040. The transmitted captured visual information may be timestamped by the visual capturing device 1050 and/or the controller 1040.The captured visual information may subsequently be stored in thestorage component 1070.

According to some embodiments, a command to display sensor data (orderived data) and/or visual data may be received by the controller 1040from the display device and its GUI 1080. The command may be transmittedto the storage component 1070 to retrieve the relevant information. Thestorage component 1070 may therefore retrieve and transmit the requestedinformation to the controller 1040 which further causes that informationto be rendered on the display (GUI) 1080 for user manipulation thereof.

Referring now to FIG. 10B, data communication flow according to someembodiments is shown. According to some embodiments, at time t₁ sensordata is transmitted from sensor 1010 to the controller 1040. Thecontroller 1040 process the received information similar to FIG. 10A.The processed and/or the raw information received may be stored atstorage 1070 at time t₁ as they are received from the sensor 1010 orshortly after. The controller 1040, at time t₂, may determine whether atriggering event has occurred. If the controller 1040 determines thatthe triggering event has occurred, it may transmit a signal to thevisual capturing device 1050 at t₂ to capture visual information and totime stamp the captured visual information. The time stamped visual datamay be transmitted at time t₃ from the visual capturing device 1050 tothe controller 1040 and stored in the storage component 1070 (eitherdirectly without going through controller 1040 (not shown) or by goingthrough the controller (1040)) at time t₃ or shortly after.

At time t₄, additional sensor data is received from the sensor 1040 bythe controller 1040. The received information or processed and derivedinformation therefrom may be stored in the storage component 1070 attime t₄ or shortly after. The controller 1040 may determine whether aterminating triggering event has occurred. The terminating triggeringevent may be based on whether one or more sensor(s) satisfy a certaincondition(s), e.g., certain measurement reading, certain measurementreading by a group of sensors, certain measurement reading by a group ofsensors within a certain period of time, etc. The controller maytransmit a signal to terminate triggering event to the visual capturingdevice 1050 at time t₅. In other words, logging of visual capturedinformation and storage therefrom may be terminated.

In some embodiments, a command may to display sensor data (or deriveddata therefrom) and/or time stamped visual data may be received by thecontroller 1040 from the display (GUI) 1080, at time t₆. The receivedcommand may cause the controller 1040 to send a command to retrievesensor data (or data derived therefrom) and/or the captured visual datafor a certain time period, e.g., data time stamped between time t₂ andt₅, from the storage component 1070. As such, the storage component 1070may transmit the information to the controller 1040 or directly to thedisplay (GUI) 1080 (not shown). In this embodiment, the controller 1040receives and relays the information to the display (GUI) 1080 at timet₉. Accordingly, the user may view the measured sensor(s) informationand/or the visually captured information for the time period ofinterest, e.g., between time t₂ and t₅.

Referring now to FIGS. 11A and 11B, flow diagrams according to someembodiments are shown. Flow diagrams 11A and 11B describe variousprocesses that were described in FIGS. 1A-10B.

At step 1110, measured value(s) associated with an input of sensor(s) isreceived. Optionally at step 1112, the measured value(s) may be timestamped. Furthermore, optionally at step 1114, the measure value(s)(with or without their timestamp) may be stored for later retrieval.

At step, 1120, captured visual data may be optionally received.Optionally at step 1122, the captured visual data may be time stampedand optionally at step 1124 the captured visual data (with or withoutthe time tamp information) may be stored for later retrieval.

At step 1130, the measured value(s) may be rendered on a GUI.Optionally, the captured visual data may also be rendered on the GUI, atstep 1140. Optionally at step 1132, a playback device may be rendered toallow a user to control the rendition of the measured value(s) and/orthe captured visual data. At step 1150, responsive to user manipulation,the GUI renders the measured value(s) and/or the captured visual datathat can be scrolled through in time.

Referring now to FIG. 11B, at step 1110, measured value(s) associatedwith an input of sensor(s) is received. At step 1116, it is determinedwhether the measured value(s) satisfy a first set of conditions or asecond set of conditions. For example, it may be determined whether thesensor(s) readings exceed a certain threshold, is within a certainrange, sensor(s) readings within a certain proximity showing a spike inmeasured values within a certain amount of time of one another, etc.

If it is determined that the first set of conditions are satisfied,steps 1112, 1114, 1122, and 1124 may be performed. If it is determinedthat the second set of conditions are satisfied, e.g., sensor(s)readings went from elevated reading to normal, sensor(s) readings thatwere reading above a certain threshold value decreased by a certainpercentage within a certain amount of time of one another, etc., steps1112 and 1114 may be performed. Furthermore, in response to the secondset of conditions being satisfied, at step 1126 the time stamping of thecaptured visual data may be terminated and at step 1128 the storage ofthe captured visual data may be stopped and terminated.

Referring now to FIG. 12, a block diagram of a computer system inaccordance with some embodiments is shown. With reference to FIG. 12, anexemplary system module for implementing embodiments includes a generalpurpose computing system environment, such as computing systemenvironment 1200. Computing system environment 1200 may include, but isnot limited to, servers, switches, routers, desktop computers, laptops,tablets, mobile devices, and smartphones. In its most basicconfiguration, computing system environment 1200 typically includes atleast one processing unit 1202 and computer readable storage medium1204. Depending on the exact configuration and type of computing systemenvironment, computer readable storage medium 1204 may be volatile (suchas RAM), non-volatile (such as ROM, flash memory, etc.) or somecombination of the two. Portions of computer readable storage medium1204 when executed facilitate the rendition of measured values and thecaptured visual data (e.g., process 1100 and as described in FIGS.1A-11B).

Additionally, in various embodiments, computing system environment 1200may also have other features/functionality. For example, computingsystem environment 1200 may also include additional storage (removableand/or non-removable) including, but not limited to, magnetic or opticaldisks or tape. Such additional storage is illustrated by removablestorage 1208 and non-removable storage 1210. Computer storage mediaincludes volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Computer readable medium 1204, removable storage 1208 andnonremovable storage 1210 are all examples of computer storage media.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, expandable memory(e.g., USB sticks, compact flash cards, SD cards), CD-ROM, digitalversatile disks (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing system environment 1200. Any suchcomputer storage media may be part of computing system environment 1200.

In some embodiments, computing system environment 1200 may also containcommunications connection(s) 1212 that allow it to communicate withother devices. Communications connection(s) 1212 is an example ofcommunication media. Communication media typically embodies computerreadable instructions, data structures, program modules or other data ina modulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. The term computer readable media as used herein includesboth storage media and communication media.

Communications connection(s) 1212 may allow computing system environment1200 to communicate over various networks types including, but notlimited to, fibre channel, small computer system interface (SCSI),Bluetooth, Ethernet, Wi-fi, Infrared Data Association (IrDA), Local areanetworks (LAN), Wireless Local area networks (WLAN), wide area networks(WAN) such as the internet, serial, and universal serial bus (USB). Itis appreciated the various network types that communicationconnection(s) 1212 connect to may run a plurality of network protocolsincluding, but not limited to, transmission control protocol (TCP), userdatagram protocol (UDP), IP, real-time transport protocol (RTP),real-time transport control protocol (RTCP), file transfer protocol(FTP), and hypertext transfer protocol (HTTP).

In further embodiments, computing system environment 1200 may also haveinput device(s) 1214 such as keyboard, mouse, a terminal or terminalemulator (either connected or remotely accessible via telnet, SSH, http,SSL, etc.), pen, voice input device, touch input device, remote control,etc. Output device(s) 1216 such as a display, a terminal or terminalemulator (either connected or remotely accessible via telnet, SSH, http,SSL, etc.), speakers, light emitting diodes (LEDs), etc. may also beincluded. All these devices are well known in the art and are notdiscussed at length.

In one embodiment, computer readable storage medium 1204 includes asensor data module 1222, a visual data module 1226, a playback module1228, and a GUI module 1230. The sensor data module 1222 is operable toreceive the measured information from sensors, process them, e.g.,determine whether a condition is satisfied, time stamp, etc. The visualdata module 1228 may be used to receive captured visual data from thevisual capturing devices and to store them (automatically or in responseto a certain condition being satisfied). The playback module 1226operates to enable the user to manipulate the rendition of measuredvalue(s) and/or the captured visual data via the GUI module 1230, and asdescribed in FIGS. 1-11B.

It is appreciated that implementations according to embodiments aredescribed with respect to a computer system are merely exemplary and notintended to limit the scope. For example, embodiments may be implementedon devices such as switches and routers, which may contain applicationspecific integrated circuits (ASICs), field programmable gate arrays(FPGAs), etc. It is appreciated that these devices may include acomputer readable medium for storing instructions for implementingmethods according to flow diagram 1100.

Referring now to FIG. 13, a block diagram of a computer system inaccordance with some embodiments is shown. FIG. 13 depicts a blockdiagram of a computer system 1310 suitable for implementing the presentdisclosure. Computer system 1310 includes a bus 1312 which interconnectsmajor subsystems of computer system 1310, such as a central processor1314, a system memory 1317 (typically RAM, but which may also includeROM, flash RAM, or the like), an input/output controller 1318, anexternal audio device, such as a speaker system 1320 via an audio outputinterface 1322, an external device, such as a display screen 1324 viadisplay adapter 1326, serial ports 1328 and 1330, a keyboard 1332(interfaced with a keyboard controller 1333), a storage interface 1334,a floppy disk drive 1337 operative to receive a floppy disk 1338, a hostbus adapter (HBA) interface card 1335A operative to connect with a FibreChannel network 1390, a host bus adapter (HBA) interface card 1335Boperative to connect to a SCSI bus 1339, and an optical disk drive 1340operative to receive an optical disk 1342. Also included are a mouse1346 (or other point-and-click device, coupled to bus 1312 via serialport 1328), a modem 1347 (coupled to bus 1312 via serial port 1330), anda network interface 1348 (coupled directly to bus 1312). It isappreciated that the network interface 1348 may include one or moreEthernet ports, wireless local area network (WLAN) interfaces, etc., butare not limited thereto. System memory 1317 includes a sensor baseddetection module 1350 which is operable to manage sensor(s) and visualcapturing device(s), and to manage the content thereof and to renderrelevant information on a GUI with a playback device for usermanipulation thereof. According to one embodiment, the sensor baseddetection module 1350 may include other modules for carrying out varioustasks. For example, the sensor based detection module 1350 may includethe sensor data module 1222, a visual data module 1226, a playbackmodule 1228, and a GUI module 1230, as discussed with respect to FIG. 12above. It is appreciated that the sensor based detection module 1350 maybe located anywhere in the system and is not limited to the systemmemory 1317. As such, residing of the sensor based detection module 1350within the system memory 1317 is merely exemplary and not intended tolimit the scope. For example, parts of the sensor based detection module1350 may reside within the central processor 1314 and/or the networkinterface 1348 but are not limited thereto.

Bus 1312 allows data communication between central processor 1314 andsystem memory 1317, which may include read-only memory (ROM) or flashmemory (neither shown), and random access memory (RAM) (not shown), aspreviously noted. The RAM is generally the main memory into which theoperating system and application programs are loaded. The ROM or flashmemory can contain, among other code, the Basic Input-Output system(BIOS) which controls basic hardware operation such as the interactionwith peripheral components. Applications resident with computer system1310 are generally stored on and accessed via a computer readablemedium, such as a hard disk drive (e.g., fixed disk 1344), an opticaldrive (e.g., optical drive 1340), a floppy disk unit 1337, or otherstorage medium. Additionally, applications can be in the form ofelectronic signals modulated in accordance with the application and datacommunication technology when accessed via network modem 1347 orinterface 1348.

Storage interface 1334, as with the other storage interfaces of computersystem 1310, can connect to a standard computer readable medium forstorage and/or retrieval of information, such as a fixed disk drive1344. Fixed disk drive 1344 may be a part of computer system 1310 or maybe separate and accessed through other interface systems. Networkinterface 1348 may provide multiple connections to other devices.Furthermore, modem 1347 may provide a direct connection to a remoteserver via a telephone link or to the Internet via an internet serviceprovider (ISP). Network interface 1348 may provide one or moreconnection to a data network, which may include any number of networkeddevices. It is appreciated that the connections via the networkinterface 1348 may be via a direct connection to a remote server via adirect network link to the Internet via a POP (point of presence).Network interface 1348 may provide such connection using wirelesstechniques, including digital cellular telephone connection, CellularDigital Packet Data (CDPD) connection, digital satellite data connectionor the like.

Many other devices or subsystems (not shown) may be connected in asimilar manner (e.g., document scanners, digital cameras and so on).Conversely, all of the devices shown in FIG. 13 need not be present topractice the present disclosure. The devices and subsystems can beinterconnected in different ways from that shown in FIG. 13. Theoperation of a computer system such as that shown in FIG. 13 is readilyknown in the art and is not discussed in detail in this application.Code to implement the present disclosure can be stored incomputer-readable storage media such as one or more of system memory1317, fixed disk 1344, optical disk 1342, or floppy disk 1338. Theoperating system provided on computer system 1310 may be MS-DOS®,MS-WINDOWS®, OS/2®, UNIX®, Linux®, or any other operating system.

Moreover, regarding the signals described herein, those skilled in theart will recognize that a signal can be directly transmitted from afirst block to a second block, or a signal can be modified (e.g.,amplified, attenuated, delayed, latched, buffered, inverted, filtered,or otherwise modified) between the blocks. Although the signals of theabove described embodiment are characterized as transmitted from oneblock to the next, other embodiments of the present disclosure mayinclude modified signals in place of such directly transmitted signalsas long as the informational and/or functional aspect of the signal istransmitted between blocks. To some extent, a signal input at a secondblock can be conceptualized as a second signal derived from a firstsignal output from a first block due to physical limitations of thecircuitry involved (e.g., there will inevitably be some attenuation anddelay). Therefore, as used herein, a second signal derived from a firstsignal includes the first signal or any modifications to the firstsignal, whether due to circuit limitations or due to passage throughother circuit elements which do not change the informational and/orfinal functional aspect of the first signal.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the claimed embodiments to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings.

What is claimed is:
 1. A system comprising: a sensor configured tomeasure a value associated with an input; a visual capturing deviceconfigured to capture visual data; and a controller configured to causethe measured value and the captured visual data to be rendered on adisplay device simultaneously.
 2. The system as described in claim 1,wherein the sensor is selected from a group consisting of thermalsensor, electromagnetic sensor, mechanical sensor, motion sensor, andbiological/chemical sensor.
 3. The system as described in claim 1,wherein the visual capturing device is selected from a group consistingof a camera and a video camera.
 4. The system as described in claim 1,wherein the controller is further configured to cause the measured valueto be time stamped and stored in a storage medium for later retrieval.5. The system as described in claim 1, wherein the controller is furtherconfigured to determine whether the measured value satisfies a certaincondition.
 6. The system as described in claim 5, wherein the controllerin response to determining that the measured value satisfies the certaincondition causes the captured visual data to be time stamped and storedin a storage medium for later retrieval.
 7. The system as described inclaim 6, wherein the sensor and the visual capturing device are within asame physical proximity, and wherein the captured visual data isassociated with the measured value.
 8. The system as descried in claim1, wherein the captured visual data is time stamped and stored in astorage medium for later retrieval.
 9. The system as described in claim1, wherein the sensor and the visual capturing device are within a samephysical proximity, and wherein the captured visual data is associatedwith the measured value.
 10. The system as described in claim 1, whereinthe sensor is configured to measure a plurality of values associatedwith the input, and wherein the visual capturing device is configured tocapture a sequence of visual data associated with the plurality ofvalues, and wherein the controller is configured to cause the pluralityof values and the sequence of visual data to be rendered on the displaydevice simultaneously and scroll through in time responsive to usermanipulation.
 11. The system as described in claim 1, wherein thecontroller is further configured to display a representation of thesensor on a map, wherein the representation of the sensor on the map ispositioned in its geo-positional location on the map.
 12. A systemcomprising: a plurality of sensors configured to measure valuesassociated with inputs associated therewith; a plurality of visualcapturing devices configured to capture visual data associatedtherewith, wherein a subset of the plurality of sensors is associatedwith a subset of the plurality of visual capturing devices; and acontroller configured to cause measured values associated with thesubset of the plurality of sensors that is further associated withcaptured visual data associated with the subset of the plurality ofvisual capturing devices to be simultaneously rendered on a displaydevice.
 13. The system as described in claim 12, wherein the subset ofthe plurality of sensors is associated with the subset of the pluralityof visual capturing devices in response to a user input.
 14. The systemas described in claim 12, wherein the subset of the plurality of sensorsis associated with the subset of the plurality of visual capturingdevices in response to heuristics.
 15. The system as described in claim14, wherein the heuristics is based on latitude and longitude of thesubset of the plurality of sensors and further based on latitude andlongitude of the subset of plurality of visual capturing devices, andwherein the heuristics is further based on whether the subset of theplurality of sensors and the subset of the plurality of visual capturingdevices are within a same physical boundaries such that the subset ofplurality of visual capturing devices is configured to provide visualqueues associated with physical proximity of the subset of the pluralityof sensors.
 16. The system as described in claim 12, wherein the subsetof the plurality of sensors associated with the subset of the pluralityof visual capturing devices share a similar geo-locationalcharacteristic and are within a certain physical proximity of oneanother.
 17. The system as described in claim 12, wherein the controlleris configured to cause the measured values associated with the subset ofthe plurality of sensors to be time stamped and stored, and wherein thecontroller is further configured to cause a subset of the capturedvisual data associated with the subset of the plurality of visualcapturing devices to be time stamped and stored responsive to themeasured values associated with the subset of the plurality of sensorssatisfying a certain condition.
 18. A method comprising: receiving aplurality of measured values from a sensor; receiving a plurality ofcaptured visual data from a visual capturing device, wherein the visualcapturing device is associated with the sensor, and wherein theplurality of captured visual data is associated with the plurality ofmeasured values; and rendering the plurality of measured values and theplurality of captured visual data simultaneously on a graphical userinterface.
 19. The method as described in claim 18 further comprising:determining whether a measured value from the plurality of measuredvalues satisfies a certain condition; and responsive to determining thatthe measured value from the plurality of measured values satisfies thecertain condition, time stamping and storing captured visual data thatis associated with the measured value that satisfies the certaincondition and further time stamping and storing captured visual datasubsequent to the captured visual data that correspond to the measuredvalue that satisfies the certain condition.
 20. The method as describedin claim 19 further comprising: determining whether another measuredvalue of the plurality of measured values does not satisfy the certaincondition, wherein the another measured value is at a time subsequent tothe measured value that satisfies the certain condition; responsive todetermining that the another measured value does not satisfy the certaincondition, terminating the time stamping and the storing of capturedvisual data that are associated with a time subsequent to the anothermeasured value that does not satisfy the certain condition; andresponsive to determining that the another measured value does satisfythe certain condition, maintaining the time stamping and the storing ofcaptured visual data that are associated with a time subsequent to theanother measured value that do satisfy the certain condition.
 21. Themethod as described in claim 18 further comprising: scrolling in timethrough the plurality of measured values and the plurality of capturedvisual data simultaneously on the graphical user interface in responseto a user interaction with the graphical user interface.
 22. The methodas described in claim 18, wherein the sensor and the visual capturingdevice are within a same physical boundaries such that the visualcapturing device is configured to provide visual queues associated withphysical proximity of the sensor.